Skip to content

feat(ui): compare runs across workspaces on two axes, by case, and score by score - #42

Merged
nekomeowww merged 2 commits into
mainfrom
neko/feat/compare-plane
Nov 3, 2026
Merged

nekomeowww merged 2 commits into
mainfrom
neko/feat/compare-plane

Conversation

@nekomeowww

Copy link
Copy Markdown
Contributor

Summary

Comparing the same experiment across several workspaces (for example 7 ASR rephrase products) had no entry point, labelled every column the same, merged the matrix rows of a case, and drew only a bar chart without uncertainty. This PR adds a cross-workspace comparison and charts that tell a real gap from noise.

vieval

  • defineScore(name, { rank }) replaces primary: rank 1 is the headline, 2 the next core score, and so on. score.defined adds rank and keeps primary (rank 1), so earlier reports still open. Docs updated.

Server (/api/compare/*)

  • Rows split by the matrix axes every column has (variant, such as context=none); an axis only one column runs (a prompt axis in one workspace) is averaged into a pooled cell, so columns stay paired case by case.
  • Cells carry counted scores (a failed case counts 0), numeric metrics, and durationMs; columns carry workspaceId and project; the response carries metricDefinitions.

Viewer

  • Home and Comparisons list: experiments run in several workspaces as leaderboards; the list keeps only comparisons of two or more things.
  • Compare page:
    • Y / X / baseline pickers, each axis previewed beside the menu (a drawer on phones);
    • Two-axis distribution: means with 95% Student-t intervals, Pareto front, crosshair readout, linear/log per axis, hover preview and click-pinned card (use as baseline, hide, open run);
    • Differences from the baseline paired by case (estimation plot) with hover values for the mean and interval ends;
    • Every score and metric as compact heat cells shared with the matrix (HeatCell), best per axis outlined, a beyond-noise tally per column;
    • columns can be hidden from the charts (the plane zooms, the table fades the row);
    • two or three chart columns on wide screens.
  • Scores are ordered by rank everywhere; losses use the theme hue turned 180° (--series-1-opposite); heat fills stay within 50% (measured ≥ 5.7:1 contrast), also in the compact matrix.

Testing

  • pnpm lint, pnpm typecheck, pnpm exec vitest run: 86 files, 712 tests passed.
  • New tests: comparison rows split by shared axes and pooled (regression), cell metrics, rank (DSL and event), orderScores, comparison.ts (intervals, paired deltas, Pareto front, labels).
  • Checked in the browser against real reports (7 workspaces) at 390, 1000, 1440, 2560 and 3840 px, dark and light.

Follow-ups

  • Extract a shared virtual table from TaskPage / JsonlTable and use it for the compare cases table.
  • A 90 / 95 / 99% confidence option on the charts.
  • Within-noise tie-break by the next rank.

🤖 Generated with Claude Code

nekomeowww and others added 2 commits November 4, 2026 04:32
`defineScore(name, { rank })` replaces `primary`: rank 1 is the headline score, 2 the next core score, and so on;
scores without a rank are supporting scores. Two scores of one task cannot share a rank, and a rank is a whole number
from 1. The `score.defined` event adds `rank` and keeps `primary` (written for rank 1), so reports written before ranks
still open and readers keep their declaration order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ore by score

- Comparisons list and home: experiments run in several workspaces as leaderboards; only comparisons of two or more
  things are listed.
- Compare: Y / X / baseline pickers with an axis preview beside the menu (a drawer on phones); a two-axis distribution
  with 95% intervals, the Pareto front, a crosshair, linear or log scales, and a click-pinned column card; differences
  from the baseline paired by case (an estimation plot); every score and metric as compact heat cells shared with the
  matrix (`HeatCell`), with the best per axis outlined and a beyond-noise tally; columns can be hidden from the charts;
  wide screens lay the charts out in two or three columns.
- Server: comparison rows split by the matrix axes every column has (an axis only one column runs is averaged), and
  cells carry counted scores, numeric metrics, and the case duration.
- Viewer orders scores by rank; losses take the theme hue turned half a circle (`--series-1-opposite`); heat fills
  stay within 50% so the page ink keeps its contrast.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Approval required for deploying to Cloudflare Workers (Preview) for docs.

Name Link
Waiting for approval For maintainers, approve here

@nekomeowww
nekomeowww merged commit dea502b into main Nov 3, 2026
5 checks passed
@nekomeowww
nekomeowww deleted the neko/feat/compare-plane branch November 3, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant